Thanks - bringing back the delimiter is fine by me (although I don't fully understand what this is doing that is causing this speed-up).

One thing - when I run this locally I get an error:

set s3_region='us-east-2';
FROM glob('s3://coiled-datasets/timeseries/*-years/parquet/*.parquet');

HTTP Error:
HTTP GET error reading 's3://coiled-datasets/timeserieshttps://coiled-datasets.s3.us-east-2.amazonaws.com/?encoding-type=url&list-type=2&prefix=timeseries%2F&delimiter=%2F' in region 'us-east-2' (HTTP 403 Forbidden)

Authentication Failure - this is usually caused by invalid or missing credentials.
* Credentials are provided, but they did not work.
* See https://duckdb.org/docs/stable/extensions/httpfs/s3api.html

LINE 1: FROM glob('s3://coiled-datasets/timeseries/*-years/parquet/*...
             ^

There seems to be something going wrong with concatenation of URLs (s3://coiled-datasets/timeserieshttps://...). Are you seeing this locally as well?

I don't reproduce, this smells like you might have credentials or region enabled? That it's a longstanding (and weird) issue.

Can you check SELECT current_setting('s3_access_key_id'); (and family)?

As to why this is faster, basically in the case of multiple folders, like:

my-dir/file1.parquet
my-dir/file2.parquet
other-dir/a/file.parquet
other-dir/a/file2.parquet
zz-dir/z/file2.parquet
other-folder/a/file2.parquet

Say that you glob on *-dir/a/*.parquet, before the PR it would list all files in the bucket, and then select 3rd and 4th.
After it would:

1. list everything up to the delimiter (so getting back dir/ + other-folder/)
2. iterating on my-dir, that matches, now listing on my-dir/a/*.parquet will get 0 files.
3. iterationg on other-dir, that matches, now it should list every file in dir/a/*.parquet, getting 2 files.
4. iterating on zz-dir, that matches
5. iterating on other-folder, that do not matches

Now that I write it down, it can be more expensive since every level gets an extra listing, advantage it should be that those are parallelizable, while the single full folder scan it's not parallelizable.